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STATEMENT OF THE PROBLEM 

B. P. Skinner's application of his principles of learning 
to teaching machines and auto-instructlon has resulted In con- 
siderable debate as well as research on response characteristics 
associated with these devices, Skinner's views are based on 
his concept of "shaping" in which gradual changes In response 
are reinforced. He states that in the delicate process of 
shaping behavior via program instruction, incorrect responses 
should be minimized because such responses may be learned 
(Skinner, 1958). Multiple-choice material which contains 
plausible wrong responses would tend to promote incorrect 
responses which would be learned, Skinner is supported in 
this point of view by Porter (1957). 

AS a result of this conception both Skinner and Porter 
have been instrumental in the promotion of programing known 
as linear programing in which errors in response are held to 
a minimum by maximizing cueing for each frame within the pro- 
gram. Needless to say, multiple-choice options are virtually 
eliminated from this type of program, 

Pressey (1962) has argued that multiple-choice alterna- 
tives should be employed because exposure to wrong alternatives 
may help to clarify or delimit the meaning of the correct 
response thereby aiding in the process of developing discrimi- 
nations, He further suggests that the introduction of plausible 
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alternatives provides a more challenging leai ilng task as 
compared to a program In which few errors are possible. 

However, the programs developed by Pressey are not pro- 
grams In the same sense as linear programs. The Items selected 
for Inclusion are not sequenced nor are they interdependent 
and interlocking as In linear programs of the type espoused 
by Skinner. Pressey >s approach seems to be a "testing" approach 
as opposed to a learning approach, 

Pressey »s different conception of learning has led him 
to defend the use of multiple-choice Items in auto-lnstructlon 
situations. But some further Investigation of the use of 
multiple-choice responses In the linear programs seems In order. 

In addition to the Issue stated above, there has been 
considerable research on the problem of response mode. Should 
the response be overt and constructed or does covert responding 
or the Indicated response (as In the case of Indicating one 
of the multiple-choice alternatives as correct) function 
equally well? The evidence to date tends to support some form 
of overt responding, but differences between overt and indi- 
cated responses and overt and constructed responses have not 
been investigated without confounding the effects of multiple- 
choice items with the Indicated response and the constructed 
response with a recall type of frame. In this study some 
subjects will be required to write the alternative they 

select as correct while others will Indicate the alternative 
they select. 



Little information Is available on the effects of overt 
correction of errors In programed instruction. The information 
that is available suggests that the correction procedure holds 
some promise. In view of Guthrie's recency principle it would 
appear that the correction of errors by overtly responding 
after learning of one's mistakes would tend to counteract the 
effects of making the error in the first place. If plausible 
wrong alternatives somehow prompt an error, then overtly 
responding with the correct answer after learning of that 
error would seem to be a sound technique for correction. 

Finally, the individual and combined effect of the response 
characteristics on the repetition of errors from immediate 
tests to delayed tests will be studied. In particular the 
effect of the plausible wrong alternatives of the multiple- 
choice frames on errors will be noted. In order for these 
kinds of analyses to be made, the criterion tests will be 
composed of selected frames and part-frames of the learning 
program. 

The purpose of this study is to simultaneously investi- 
gate the effects of easy versus difficult multiple-choice 
alternatives, the written response versus the Indicated 
response, and the correction procedure versus the non-correction 
procedure which allows for additional analysis of interaction 
effects among these variables. 
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Objectives 

The major objectives of this study are to determine the 
effects of the response characteristics of difficulty of alter- 
natives, response mode, and correction procedures on a variety 
of criterion measures* For some oi the above response vari- 
ables earlier research or Intuition suggests the direction of 
the results when particular criterion measures are employed. 

In these Instances, this Investigator can make some state- 
ment as to expected outcomes. There are several other effects 
to be Investigated, however, for which statements about 
expected outcomes are meaningless because of the paucity of 
data relative to these results. This Is particularly true 
of the double and triple Interactions which are by-products 
of a three-way analysis of variance. 

Following are the objectives with some discussion about 
expected outcome where such discussion seems warranted. 

To Investigate the effects of the response characteristics 
of difficulty of alternatives, response mode, and correction 
procedures and their Interactions employing the following 
criterion measures: 

1) Rate of errors In responding during the course of 
the program. 

By definition one would expect more errors to be 
associated with the difficult alternatives. One would 
also expect more errors to be associated with the dif- 
ficult alternatives, non-correction procedure. The 
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remaining effects are difficult to predict although one 
would not be too surprised if the correction procedure 
resulted in fewer errors. 

2) Errors on a test of immediate retention employing a 
recall, constructed response test. 

It is predicted that those Ss who are required to 
construct a response will make fewer errors than those 
who indicate their response on this criterion measure. 
Beyond this statement it is difficult to predict the 
remaining results. 

3) Errors on a test of immediate retention employing a 
multiple-choice test. 

It is predicted that the difficult alternatives and 
the correction procedure will result in the least number 
of errors with the possibility of the fewest errors being 
associated with the difficult alternative, correction 
procedure combination. Past investigations have shown 
little advantage accruing to a particular mode of response 
when a multiple-choice type of item is employed. 

4) Errors on a test of delayed retention employing the 
recall, constructed response test. 

It appears likely that whatever effects may be 
evident immediately after the learning situation may, 
in part, be dissipated over time. A paucity of research 
data on this question makes prediction difficult. 
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5 ) Errors on a t€ist of delayed retention employing a 
multiple-choice test. 

It Is not possible to predict the outcome, 

6 ) Items missed on both of the test positions of Imme- 
diate and delayed retention employing a) the recall, 

constructed response test, and b) the multiple-choice 
test. 

According to the position taken by Skinner and Holland 
we would expect the largest number of errors to be asso- 
ciated with difficult alternatives, particularly when 
employing the multiple-choice test. This statement 
assumes that more errors will be made on the program v;lth 

the difficult foils appended and that these errors will 
endure, 

7 ) Response errors which are repeated on both of the 
test positions of immediate and delayed retention 
employing a) the recall, constructed response test, 
and b) the multiple-choice test. 

As per the above comment one would expect a greater 
number of response error repetitions to be associated 
with the difficult alternatives. 

Related Research 

Programed Instruction has stimulated a great deal of 
research of questionable value as well as some of value. 

In a summary of the research on mode of response Alter and 
Silverman (1963) suggest that the ease in manipulation of 



this variable resulted in a large number of investigations, 
some of which were done rather quickly and poorly with insuf- 
ficient safeguards including length of program, size of sample, 
and suitability of the criterion measures. Holland (1965 j 
p. 108) asserts that: 

With a fev; exceptions the data are collected 
under poorly controlled conditions. Programed books 
are often used, and these offer little protection 
from a variety of distortions of the data including 
outright cheating by the subjects. Often, the work 
is done in open classes of 20 to 30 students, in 
many cases, without the supervision of a trained 
experimentalist . 

The present study was done with the hope that many of the 
weaknesses of the past research in this area will be eliminated, 
thus allowing for more precise information. 



Multiple-Choice versus the Constructed Response 
The most consistent finding of those investigators 
(Burton and Goldbeck, 1962; Coulson and Silberman, I960; 

Pry, i960; Hough, 1962; Moore and Smith, 1964; Williams, 1965) 
who have attempted to directly compare multiple-choice response 
and constructed response on verbal programs has been that of 
no differences. The major exception was Pry's ^1960) study 
which showed a significant difference in favor of the con- 
structed response tested by recall. 

Questions left unanswered by these investigations are many. 
The majority of these researchers did not specify the procedure 
for selecting the multiple-choice alternatives. Are we to 






assume that alternatives were selected to maximize difficulty 
of the required discrimination? Were synonyms used as logical 
alternatives? Synonyms may be appropriate alternatives for 
some programs^, but inappropriate for others. For example. 

It may be that a correct response will have associated with 
it several other terms, all of which are appropriate in this 
Instance. To force discriminations among these terms would 
be inappropriate even though the result would undoubtedly be 
an Increase in the number of errors. Conversely, there are 
Instances in which synonymous terms cannot all be considered 
correct because of the technical nature of the term, and to 
force discriminations would be appropriate and would also 
result In a greater number of errors. It is the contention 
of this investigator that the method of selection of the 
alternatives will greatly affect the results. Burton and 
Ooldbeck (1962) attempted to research the effects of easy and 
difficult multiple-choice alternatives, but the study has 
been rather severely criticized on other grounds, (Holland, 1965) 
thus rendering the results questionable in value. 

The nature of the criterion Instrument (s) was not clear 
in the majority of the above studies. There is little doubt 
that the selection of Items which make up the criterion test 
is another crucial factor, a cortgruence between the kinds of 
discriminations programed and the kinds evaluated would likely 
result In quite a different outcome than if the criterion test 
was designed to evaluate the more general aspects of the program. 
This point is so crucial to the make-up of a sound investigation 
that to neglect it is to invite criticism. 
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In addition to these comments on previous studies^ one 
more Is In order. The Investigators cited above Invariably 
associated the constructed response made with the recall 
programing frame. Perhaps the effectiveness of the multiple- 
choice arrangement may be enhanced by requiring subjects to 
construct the response that they select. Wouldn't a comparison 
of the recall frame with a multiple-choice frame In which both 
required a constructed response provide some data of Interest? 
This Investigation will attempt to collect some data In answer 
to this question. 

Constructed versus Indicated Response 

A large number of research studies have attempted to 
answer the question of the effect of overt responding as 
compared to covert responding. Summaries of these kinds of 
studies have been written by Alter and Silverman (1962) and 
Holland (l 9 o 5 ). Generally, the results have shown no differ- 
ences, but in a few studies the overt responding technique 
was superior. Holland, assuming that the overt response Is 
more effective, points out that a post-test advantage for 
overt responding exists when the answers are contingent on 
the Important content, the error rate Is low, and when programs 
are of sufficient length so as to Increase the likelihood that 
the covert responder will be lulled Into not responding at all. 

In this investigation the major question In regard to 
response mode is whether there is a difference between two 
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forms of overt response, that of writing the answer or of 
Indicating the answer. Little evidence Is available In regard 
to this comparison except In those Instances In which mode of 
overt responding Is confounded with the Issue of multiple-choice 
versus constructed response. 

Correction versus Non-Correction 

There Is meager research evidence on the effects of 
correction of a response after feedback. The majority of 
studies have focused on the effects of feedback without cor- 
rection of the response. Irion and Briggs (1957) found that 
correction of the wrong response before proceeding to the next 
frame resulted In superior performance as compared to a non- 
correction procedure. However, the correction procedure was 
Inferior to a prompting procedure In which the subject was 
given a chance to observe the correct response before responding, 
thus eliminating the possibility of error. In this Instance It 
appears that errors tended to Interfere with the acquisition 
process. Little mention was made of the type of learning 
program used. It Is doubtful that the response was determinate 
or that the program was anything but a series pf rather 
dlsconnectfd^^tems f^jiges.. 

Another prompting, variation was^plso superior to the 
correction procedure. In this approach the subject makes one 
choice after which he Is told that he Is right or wrong, if 
he Is wrong the correct response Is Indicated and the subject 
then makes the correct response. Here the possibility of 



making errors Is greatly reduced as compared to the correction 
procedure In which the subject continues to respond until he 
makes the correct answer, Irion and Briggs speculate that the 
fact that the stimulus and correct response appeared in close 
temporal contiguity may be one of the reasons for the success 
of the prompting techniques. 

Suppes and Ginsberg (1962) found that a correction procedure 
was more effective than a non-correction procedure for children 
attempting to learn arable number responses to limited binary 
number stimuli. The same results did not obtain for adults 
(Burke, Estes, and Hellyer, 1954). 

Glaser (1965) discusses the Issue of correction in conjunc- 
tion with the problem of errors. He states that the concept of 
correction has received little attention In the literature 
and Is a topic greatly in need of research. 

It appears that any Investigation of the effects of a 
correction procedure and learning must, of necessity, be 
Investigated In relationship to a number of additional vari- 
ables, For example, Kaess and Zeaman (i960) found that the 
greater the number of plausible alternatives the greater the 
number of errors. Again, some question Is raised about the 
type of learning program employed which In this case appears 
to be composed of a series of disconnected Items more on the 
order of a Pressey-type program. Whether these same results 
would obtain If a linear, low error rate program were employed 
is a question open to research. 
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The present study will use a linear program which mini- 
mizes error rate and four multiple-choice alternatives in an 
attempt to investigate the effect of correction and non-correction 
on programed instruction. 



CHAPTER II 



PROCEDURE 

Subjects 

Two hundred subjects from an introductory educational 
psychology course were randomly assigned Into the experimental 
groups resulting In 25 Ss per cell. Participation in the 
experimental sessions was a course requirement , The teachers 
of these classes explained that performance In the experiment 
would not influence the evaluation of their course performance. 
The nature of the experiment was briefly explained and Ss were 
told that the concepts they learned during the experiment would 
be of value later in the course. 

The Program 

Six of the first eight sets (1, 2, 3> Is 8) of Holland 
and Skinner's Analysis of Behavior (196I) were converted to 
a multiple-choice format, resulting in a total of I86 frames 
and 233 responses*^ The Holland-Sklnner program was selected 
because It is suitable for the available population, and 
evidence suggests that it is a well -designed program. 

A small number of frames In each set were modified so that 
the central concept under consideration would appear in all 

I 

frames as the response rather than as part of the stimuli. 

^Permission granted by McGraw-Hill Book Company. 




14 



The purpose of these modifications was to facilitate the 
study of error repetitions by placing the concept itself 
rather than a functional term in the response position. 

Concurrent with the above modifications^ three difficult 
and three easy multlple-cholce foils were appended to each 
frame. These alternatives plus the correct response resulted 
in four multiple-choice alternatives per frame. The difficult 
alternatives were derived by selecting synonyms to the correct 
response and by choosing from among similar psychological 
concepts which appeared on frames approximately contiguously 
with the frame in question. It was felt that the choice of 
‘‘technical" concepts highly related in meaning and encountered 
close in time to the correct response of a given frame would 
afford maximal difficulty by forcing discriminations among 
relatively unfamiliar and yet technical concepts. Examples 
of the kinds of alternatives developed may be found in Appendix A, 

Easy multiple-choice alternatives were generated by 
selecting non-psychological synonyms to a psychological concept 
highly dissimilar to the correct response. Thus the task of 
discriminating among alternatives here would be easily achieved 
on either or both of the factors of the non-psychological 

nature of the incorrect alternatives and/or on their dissimi- 
larity of meaning. 

The second independent variable under investigation was 
mode of response. Those subjects completing their programs 
under the construct mode of response were required to write 
the response they selected while subjects under the indicate 

condition were required to indicate their choice by writing 
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the letter corresponding to the response they chose. The 
programs were identical in format, but different instructions 
were given to vary the mode of response to be effected by 
the Ss. 

The third and last independent variable under investigation 
was correction or non -correct ion of program errors. For the 
non -correct ion condition the subject responded to a frame 
after which the correct response was revealed; he then advanced 
to the next frame without an overt correction response required 
in the event of an error. However, for the correction condition, 
teaching machines were equipped with a second window cut in 
the display portion of the machine through which S was required 
to make an overt correction for all response errors made on 
the program. Corrections were made in the mode of response 
corresponding to the mode used in making the initial responses. 
That is, those taking the program under a constructed response 
condition were required to write out their corrections while 
those under the indicate condition corrected by writing the 
letter of the correct choice. 

The Experimental Session 

Prom five to 13 subjects participated in an experimental 
session with all sessions completed within a 17-day period from 
February l4 to March 5 # 1966. Subjects were randomly assigned 
to treatment conditions in the order of their arrival at the 
experimental sessions. General instructions were read to all 
Ss at the beginning of the experimental sessions. They were 
as follows; 
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The experiment in which you are participating deals with 
methods of presenting programed Instruction, The material which 
you will study today deals v^lth some basic principles of learning 
and Is a part of the material which will be covered In your 
educational psychology course. 

Look now to the top of your machines. In the large window 
across the top, you see the statement, "A doctor taps your knee 
(patellar tendon) with a rubber hammer to test yoiar ? ," 

You complete this statement by Indicating your answer In the 
blank space provided. 

Look now at the instructions taped to your machine (PAUSE), 
These are your specific instructions on how to work through 
the program. Now, following your Instructions, answer the 
first question In tne blank space on the right, (PAUSE), 

Now turn either knob and advance the paper about an Inch, 

The right answer will appear from behind the screen while your 
answer goes up under the glass. Check your answer with the 
right answer. 

Now on some of your machines a second space has been cut 
on the left. If your answer was wrong AND you have this second 
space, correct your error according to the Instructions on the 
machine, if you do not have this second space, these last 
Instructions do not apply to you. Are there any questions? (PAUSE), 

Once again, follow the Instructions taped to your machines 
exactly in working through the program. If you have any question 
or your machine Isn't working properly, raise your hand. Work 
at your own speed. You will not all finish at the same time 
since the programs differ, but you will probably spend about 
an hour and one-half to two hours on the material. When you 
finish, there will be a short test over some of the concepts 
covered In the program. Are there any questions? (PAUSE), 

Begin now on Item two. 

Individual Instructions particular to error correction and 
response mode conditions were taped to the teaching machines 
appropriately loaded for the combination of conditions desired 
and Ss' attention directed to these Instructions through the 
general Instructions previously presented. Instructions for 
specific treatment conditions were as follows: 



Difficult Construct Correct and Easy Construct Correct 

In working through the program, WRITE OUT your answers 

COMPLETELY in the space on the right. You do not need to 
write the letter of the answer. 
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If after checking your answer you see you have made an 
error, correct It by WRITING OUT COMPLETELY the correct answer 
in the space on the left. 



Difficult Indicate Correct and Easy Indicate Correct 

In working through the program, indicate your answer by 
writing the LETTER of the correct choice in the space on the 
right. Do not write the words, only indicate your choice bv 
writing A, B, C, or D. 

If after checking your answer you see you have made an 
error, correct it by writing the letter of the correct answer 
in the space on the left. Again, do not write the words; 
only indicate the correct answer by writing A, B, C, or D, 



Difficult Construct Non—Correct and Easy Construct Non—Correct 

In working through the program, WRITE OUT your answers 
COMPLETELY in the space on the right. You do not need to 
write the letter of the answer. 



Difficult Indicate Non-Correct and Easy Indicate Non-Correct 

In working through the program, indicate your answer by 
writing the LETTER of the correct choice in the space on the 
right. Do not write the words, only indicate your choice by 
writing A, B, C, or D, 

Arrows led from the instructions to the appropriate space 
on the machine. Additionally, for the four conditions involving 
non-correction, machines were not adapted to permit correction 
and no reference to the need for correction was made in these 
individual instructions, 

A starting time was recorded for each S at this point in 
the session. Twenty minutes after the beginning of the session 
exhibits associated with Sets 3, 4, and 8 were handed out to 
each subject and the following instructions were read to the 
group : 
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Later In the program, you will be Instructed by the 
program to refer to these materials. Do not read them now, 
only when you are told to read them In the program. 

As each S finished the program, the time taken to complete 
the program was recorded, and the S was required to respond 
to 54 test Items composed of 27 completion Items and 2? multiple- 
choice Items, The order of administration of the two types of 
Items was systematically varied for each S with the order of 
administration remaining the same for a given subject for both 
Immediate and delayed testing. 

Criterion Measures 

Time taken to complete the program, number of errors on 
the program, and errors made on the two-part test mentioned 
previously were the criterion measures employed. The latter 
measure was administered Immediately after the exposure to the 
program and again approximately tv/o weeks later. The delayed 
test varied as much as two days per subject because all subjects 
were tested the second time within their regular classroom 
session. Since this variation was consistent for all experi- 
mental conditions. It was felt that the results of this study 
were not biased. 

The test Items for the two types of criterion tests were 
composed of parts of program frames considered most appropriate 
as a test of each major concept presented. Part frames were 
used In order to eliminate cues designed to evoke the correct 
response. The completion Items were taken directly from the 
linear program without any further modification. The multiple- 
choice Items were constructed by using the same part frames as 
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previously discussed, but with difficult multiple-choice 
alternatives appended which were different from those used 
on the program. For the 27 program frames employed later as 
test items, a pool of six difficult multiple-choice alternatives 
were selected. Alternatives were then paired according to 
degree of difficulty and members of each pair randomly assigned 
to the program with the remaining member then assigned to the 
test. 

An attempt to analyze the effects of errors made during the 
program on errors made later on the criterion tests failed for 
lack of data. Because of the low error rate of the program 
and the fact that approximately 25 percent of the responses 
required during the program were Included on the criterion 
tests there were many Instances when the Ss did not repeat 
any of the errors on the tests that were rnade on the program. 

However, analyses of error repetition from the lmmedlaT:e 
to the delayed tests were completed. That is, cases in which 
the S missed the items on both the immediate and delayed tests, 
but made a different response each time, were recorded and used 
as a criterion measure. And finally, those cases in which the 
response was in error on the immediate tests and the same, 
wrong, response was repeated on the delayed tests were also 
recorded and used as a criterion measure. 

Treatment of the Data 

Combination of the three major Independent variables 
resulted in the following treatment conditions; 

1, Difficult Alternatlves-Constructed Responses- 
Correction (DCC) 
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2. Difficult Alternatives-Constructed Hesponses- 
Noncorrection (DCNC) 

3. Difficult Alternatives-Indicated Responses- 
Correction (DIC) 

4. Difficult Altarnatives-Indicatad Rasponses- 
Noncorrection (DING) 

5. Easy Alternatives-Constructed Resoonses- 
Correction (ECC) 

6. Easy Alternatives-Constructed Responses- 
Noncorrection (ECNC) 

7# Easy Alternatives-Indicated Hesponses- 
Correction (EIC) 

8. Easy Alternatives-Indicated Responses- 
Noncorrection (EINC) 

All criterion measures previously mentioned were serially 
employed with a 2 x 2 x 2 factorial analysis of variance 
statistical design. Main and interaction effects resulted 

from these analyses and simple effects were tested where 
appropriate. 



CHAPTER III 



RESULTS 

The same sets of the Holland-Skinner program that were 
used for the experimental groups were administered jn their 
usual format, that of the recall-construct type with feedback 
but no correction, to seventeen students. The purpose of 
this adjunct to the proposed research was to provide a basis 
for comparing a usual linear program with the various modified 
formats used in this study, and in particular to compare the 
usual linear program with the condition in which the students 
constructed answers and did not correct (dcNC), The latter 
condition was similar to the usual linear program in every 
way except for the difficult multiple-choice foils appended 
to each response position within the frame, A comparison 
between these two programs should generate data which is 
germane to the issue of the effect of multiple-choice responses 
on error production on the program and on the criterion tests 
as well. 



Descriptive Data 

The mean proportion of errors on all experimental con- 
ditions and on the usual linear program is presented in Table 1, 
As expected, those Ss exposed to the easy alternatives made 
the fewest errors on the program. The Ss exposed to the 



TABLE 1 



PROPORTION OP ERRORS TO TOTAL RESPONSES 
ON THE PROGRAM AND ON IMMEDIATE AND DELAYED COMPLETION 
AND MUIffIPLE-CHOICE CRITERION TESTS 



TREATMENT 

CONDITION 


PROGRA^M 


IMMEDIATE 

COMPIETION 


DELAYED 

CCMPIETION 


IMMEDIATE 
MULT. CHOICE 


DELAYED 
MULT. CHOICE 


DCC 


.086 


.186 


.357 


.046 


.081 


Die 


.129 


.261 


.420 


.084 


.l4l 


DCNC 


.133 


.310 


.449 


.104 


.144 


DINC 


.162 


.343 


.461 


• 

o 

VO 

VO 


.124 


ECC 


.015 


.319 


.469 


.109 


.143 


EIC 


.020 


.341 


.410 


.080 


.156 


ECNC 


.014 


.350 


.450 


.101 


.161 


EINC 


.020 


.374 


.464 


.117 


.183 


RECALL- 

CONSTRUCT 


.192 


.273 


.370 


.102- 


. Ill 
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difficult alternatives were next In error production with the 
constructed response and correction being associated with the 
fewest errors within that categoi’y. Finally, Ss responding to 
the usual linear program made the greatest number of errors 
in comparison to all conditions. The addition of multiple- 
choice alternatives does not appear to have an adverse effect 
on error production as compared to the usual linear program. 

Scrutiny of the test results reveals quite different 
findings. It Is noted that the Ss under the DCC and DIG 
conditions made the fewest errors on the completion test 
administered immediately with the usual linear program a close 
third. All other conditions yielded a larger number of errors 
on the completion test administered Immediately, It Is also 
clear that the Indicate and non-correction conditions are 
consistently associated with the larger number of errors 
regardless of the difficulty level of the alternatives. 

The results of the delayed-completlon test show that 
only the DCC condition Is superior to the usual linear approach, 
and that all other conditions generated a relatively greater 
number of errors. The results are somewhat more variable on 
this criterion measure. 

Again, the DCC condition yielded the smallest number of 
errors using the Immedlate-multlple-cholce test as the criterion 
with two conditions In which the response was indicated and 
corrected (Die and EIC) next. These results are somewhat less 
variable than those noted previously. 
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Finally, using the delayed multiple-choice test as the 
criterion It Is noted that the DCC condition Is again the most 
effective In terms of proportion of errors with the usual linear 
program second* 



Analysis 

As was mentioned earlier, the data was analyzed by 
a 2 X 2 X 2 factorial analysis of variance design. This 
analysis was repeated nine times, once for each criterion 
measure. A follow-up analysis of significant main effects is 
reported for each criterion measure. 

Program Errors 

Table 2 shows the analysis of variance for program errors 
as well as the means and standard deviations for each of the 
treatment conditions. As would be expected, the variable of 
difficulty Is highly significant with variables of response 
mode and correction being significant at the ,05 level of 
confidence. In addition, one two-way Interaction, that of 
difficulty X correction, was significant at the ,05 level, 

• 

Perhaps the most surprising finding Is that the condition 
of the response mode was significant since both forms of 
responding can be classified as overt forms. 

Inspection of the means and standard deviations suggests 
that the assumptions of normality and homogeneity of variance 
can be questioned. In this study the finding of differential 
error rates associated with the several conditions only confirms 
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TABIE 2 



ANALYSIS OP VARIANCE AND 
MEANS AND STANDARD DEVIATIONS USING 
TOTAL PROGRAM ERRORS 
AS THE CRITERION MEASURE 



SOURCE W 

VARIATION 


SllMS dff 
SQUARES 


df 


FISS 

SQUARES 


P 


DIPPICUMY 


33282.00 


1 


33282.00 


170.614 ** 


RESPONSE MODE 


1190.72 


1 


1190.72 


6.104 * 


DIFFICULTY X 
MODE 


655.22 


1 


655.22 


3.359 


CORRECTION 


1039.68 


1 


1039.68 


5.330 * 


DIFFICULTY X 
CORRECT 




1 


1142.42 


5 . 856 * 


MODE X CORRECT 


27.38 


1 


27.38 


0.l40 


DIFFICULTY X 
MODE X CORRECT 


38.72 


1 


38.72 


0.198 


MODE WITHIN 


37553.84 


192 


195.59 




TOTAL 


74829.98 


199 






* P< .05 

** p< .01 





MEANS & 


STANDARD 


DEVIATIONS OP 


CELLS 






MEAN 


S.D. 




MEAN 


S.D. 


DCC 


20.16 


9.37 


ECC 


3.64 


3.26 


Die 


30.28 


19.10 


EIC 


4.76 


4.88 


DCNC 


31.12 


18.75 


ECNC 


3.28 


3.45 


DINC 


38.00 


26.25 


EINC 


4.68 


4.61 



that the conditions, arranged on a priori grounds, do In fact 
affect error rate. In particular this Is apparent for the 
dimension of difficulty, A relatively exact test of the effect 
of these conditions on error rate Is not seen to be necessary. 

In addition, evidence In Lindquist (1953) and Scheffe (1959) 
suggests that the tests under these conditions wl]l be fairly 
accurate providing there are equal cell N*s, which Is the 
case In this study. 

An attempt was made to determine which combination of 
variables was associated with the best performance or lowest 
error rate on the program by comparing specified combinations 
of cells. For every major variable that was found to be 
significant beyond the ,05 level of significance, eight cell 
comparisons were made. For example, the following comparisons 
were made for the dimension of difficulty since that variable 
was significant ; 

A, Comparisons in which the data was collapsed over the 

correction dimension, 

1. Easy-Construct vs, Difficult-Construct 

2. Easy-Indlcate vs, Dlff icult-Indlcate 

B, Comparisons in which the data was collapsed over the 

response mode dimension, 

3. Easy-Correct vs, Difficult-Correct 

4. Easy-Noncorrect vs, Difflcult-Noncorrect 

C, Comparisons of the original treatment combinations, 

5. Easy-Construct-Correct vs, Dlff icult-Construct-Correct 

6. Easy-Indicate-Correct vs, Dlff Icult-Indlcate-Correct 

7. Easy-Construct-Noncorrect vs, Dlff Icult-Construct-Noncorrect 

8. Easy-Indlcate-Noncorrect vs, Dlff Icult-Indlcate-Ncncorrect 
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Every one of these eight comparisons shows a highly 
significant (.Ol) difference in favor of the difficult alter- 
natives of the difficulty dimension. Again, this is expected 
since the alternatives were deliberately selected to affect 
error rate. 

Later reports of the follow-ups of the significant major 
variables will include only those comparisons which yield 
significant results at the ,05 level or better. 

Only tv/o comparisons of the possible eight are significant 
in the follow-up of the significant major variable of response 
mode . The comparison of Difficult-Construct vs. Dlfflcult- 
Indicate when the data were collapsed over the correction 
dimension and DCC vs. DIG were both significant. It can be 
seen that the constructed response will result in fewer errors 
than the indicated response, especially when the alternatives 
are difficult. 

The correction dimension was a significant factor in 
reducing programing errors. The follow-up analyses showed 
that the Difficult-Correct vs. Dlff Icult-Noncorrect conditions 
:ere significantly different when the data were collapsed over 
the response mode dimension. The DCC vs. DCNC was also sig- 
nificant. Ihese results suggest that correction is an important 
factor in the reduction of errors when the a!’‘^ernatives are 
difficult and the constructed response is employed. 

Finally, the significant interaction effect between the 
difficult and correction dimensions is presented graphically 
in Figure 1. It can be seen that the factor of correction 
does not much affect the error rate when easy alternatives are 
employed. The reverse is true for the difficult alternatives. 



ERIC 



Number of program errors 
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FIGURE 1 

PLOT OF COLLAPSED MEANS 
TO SHOW INTERACTION 
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In summary of the results using program errors as the 
criterion measure, easy alternatives, constructed response, and 
correction of errors resulted In the fewest errors, but with 
the constructed response and correction of errors functioning 
best in conjunction with difficult alternatives. 

Immediate Completion Test 

The analysis of variance for the Immediate completion test 
and the means and standard deviations for each of the treatment 
conditions is presented In Table 3. Significant results were 
obtained for the dimension of difficulty at the .01 level and 
for the dimension of correction at the .05 level. The means 
Indicate that the difficult alternatives and correction activity 
resulted In the least number of errors on the completion test 
administered Immediately. 

Pollow-up comparisons of the type described before were 
executed with each of the significant dimensions. For the 
difficulty dimension, significant differences (at the .05 
level) were found between Difficult-Construct and Easy-Construct 
when the data were collapsed over the correction dimension, 
and between Difficult-Correct and Easy-Correct when the data 
were collapsed over the response mode dimension. In addition, 
the DCC vs. ECC comparison was also significant. It Is clear 
that the more difficult alternatives resulted In fewer errors 
on the completion test administered Immediately, especially 
when the difficult alternatives were combined with the con- 
structed response and/or the correction procedure. 
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TABLE 3 

ANALYSIS OP VARIANCE AND 
MEANS AND STANDARD DEVIATIONS USING 
COMPLETION TEST ERRORS 
AS THE CRITERION MEASURE 



"SOURCE (3P 

VARIATION 


SUMS DP 

SQUARES 


df 


MEa!7 

SQUARES 


P 


DIFFICULTY 


198.00500 


1 


198.00500 


7 . 4 l 8 ** 


RESPONSE MODE 


59.40500 


1 


59.40500 


2.225 


DIFFICULTY X 










MODE 


9.24500 


1 


9 . 24500 


0.346 


CORRECTION 


178.60500 


1 


178.60500 


6.691 * 


DIFFICULTY X 










CORRECT 


49.00500 


1 


49.00500 


1.836 


MODE X CORRECT 


4.20500 


1 


4.20500 


0.158 


DIFFICULTY X 










MODE X CORRECT 


4.80500 


1 


4.80500 


0.180 


MODE WITHIN 


5125.11984 


192 


26.69333 




TOTAL 


5628.39484 


199 






* p< .05 










** p< .01 













MEANS & STANDARD 
MEAN S.D. 


DEVIATIONS OF 


CELLS 

MEAN 


S.D. 


DCC 


5 . 20000 


3.20156 


ECC 


8.92000 


5.02427 


Die 


7.32000 


4.96421 


EIC 


9.56000 


5.35475 


DCNC 


8.68000 


5 . 80747 


ECNC 


9.80000 


4.51848 


DINC 


9.60000 


6.23832 


EINC 


10.48000 


5.62820 



o 
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Similarly, follow-up analyses were made with the correction 
dimension, Tt was found that Difficult-Correction conditions 
made significantly fewer errors than the Dlfflcult-Noncorrection 
condition when both of these conditions were collapsed over the 
dimension of response mode, and the Construct-Correct condition 
resulted in significantly fewer errors than the Construct- 
Noncorrect condition when the difficulty dimension was Ignored, 
DCC vs, DCNC was significant among all comparisons of Interest, 
These results suggest that the correction procedure was of value 
in error reduction when combined with difficult alternatives 
and when the response mode is the constructed response. 

Summarizing the results using the immediate completion test 
as a criterion measure, it was found that difficult alternatives 
and a correction procedure resulted in the fewest test errors. 
These factors functioned best in combination with one another 
and in conjunction with the constructed response. 

Delayed Completion Test 

The analysis of variance for the delayed completion test 
and the means and standard deviations for each of the treatment 
conditions are presented in Table 4, None of the effects 
are significant although it is noted that the DCC condition 
again resulted in the best performance. It appears that the 
effects of the variables of difficulty and correction, which 
were significant on the immediate completion test criterion, are 
somewhat diminished over time. 



o 
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TABIE ^ 



ANALYSIS OP VARIANCE AND 
MEANS AND STANDARD DEVIATIONS USING 
DELAYED COMPIETION TEST ERRORS 
AS THE CRITERION MEASURE 



'souH'CB ar 

VARIATION 


' 'SUMS Of 

SQUARES 


df 


HEaH 

SQUARES 


P 


DIPPICUDTY 


27.38000 


1 


27.38000 


1.183 


RESPONSE MODE 


2.42000 


1 


2.42000 


— 


DIPPICUHTY X 










MODE 


35.28000 


1 


35.28000 


1.524 


CORRECTION 


69.62000 


1 


69.62000 


3.007 


DIPPICUUTY X 










CORRECT 


23.12000 


1 


23.12000 


— 


MODE X CORRECT 


1.28000 


1 


1.28000 


—— 


DIPPICULTY X 










MODE X CORRECT 


36.98000 


1 


36.98000 


1.597 


MODE WITHIN 


4445.43984 


192 


23.15333 




TOTAL 


4641.51984 


199 









MEANS & STANDARD 
MEAN S.D. 


DEVIATIONS OP 


CELLS 

MEAN 


S,D. 


DCC 


10.00000 


4 t 11229 


ECC 


13.12000 


5.26245 


Die 


11.76000 


4.86724 


EIC 


11.48000 


5.10816 


DCNC 


12.56000 


4.77040 


ECNC 


12.60000 


3.84057 


DINC 


12.92000 


4.78644 


EINC 


13.00000 


5.51513 
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Imraedlate Multiple-Choice Test 
The analysis of variance and descriptive data for the 
Immediate multiple-choice test criterion are presented In 
Table 5, None of the effects are significant at the .05 level, 
but the dimension of correction was very close (3.88 where 3,90 
needed to be significant at the ,05 level). Once again It Is 

e 

noted that the DCC condition resulted In the least number of 
errors which Is not expected when a multiple-choice test Is 
the criterion. The Investigator felt that the Indicated 
response would do at least well as the constructed response 
when this criterion test was employed. 

Delayed Multiple-Choice Test 
The analyses for this delayed criterion measure and 
descriptive data are presented In Table 6, The only factor 
which was significant was that of difficulty at the ,05 level. 
Follow-up data on the variable of difficulty within the several 
comparisons previously specified failed to yield any significant 
results although all comparisons show the least number of 
errors to be associated with the difficult alternatives. Once 
again, response mode is not the powerful factor one would expect 
It to be when this type of criterion Is employed. 

Percent of Errors Made on Immediate 
Completion Test Which Were Repeated 
on Delayed Completion Test 

The data presented up to this point Is based on the 

number of errors made on either the program or immediate or 

delayed response to criterion tests, and has shown the dimensions 
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TABLE 5 

ANALYSIS OP VARIANCE AND 
MEANS AND STANDARD DEVIATIONS USING 
IMMEDIATE MULTIPLE- CHOICE TEST ERRORS 
AS THE CRITERION MEASURE 



SOURCE 73F 

VARIATION 


SUMS "OP 

SQUARES 


df 


mm 

SQUARES 


F 




DIFFICULTY 


13.52000 


1 


13.52000 


2.026 




RESPONSE MODE 


0.98000 


1 


0.98000 


0.147 




DIFFICULTY X 
MODE 


5 . 12000 


1 


5 . 12000 


0.767 




CORRECTION 


25.92000 


1 


25.92000 


3.88 




DIFFICULTY X 
CORRECT 


4.50000 


1 


4.50000 


0.674 




MODE X CORRECT 


0.00000 


1 


0.00000 






DIFFICULTY X 
MODE X CORRECT 


19.22000 


1 


19.22000 


2.880 




MODE WITHIN 




192 


6.67250 






TOTAL 


1350.37997 


199 











MEANS & 
MEAN 


STANDARD DEVIATIONS OF 

S.D. 


CELLS 

MEAN 


S.D. 


DCC 


1.28000 


1.92614 


ECC 


3.04000 


2.97881 


Die 


2.36000 


2.95635 


EIC 


2.24009 


2.24128 


DCNC 


2.92000 


3.08113 


ECNC 


2.84000 


2.33952 


DINC 


2.76000 


2.16564 


EINC 


3.28000 


2.71600 



o 
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TABLE 6 



ANALYSIS OP VARIANCE AND 
MEANS AND STANDARD DEVIATIONS USING 
DELAYED MULTIPLE-CHOICE TEST ERRORS 
AS THE CRITERION MEASURE 



SOURCE' T 5 F 
VARIATION 


SOHSTSP 

SQUARES 


df 


mm — — 

SQUARES 


P 


DIPPICUI/TY 


56.18000 


1 


56.18000 


5.1076 * 


RESPONSE MODE 


13.52000 


V 

1 


13.52000 


1.229 


Dippicunpy X 
MODE 


0.08000 


1 


0.08000 




CORRECTION 


20.48000 


1 


20.48000 


1.862 


DIPPICULTY X 
CORRECT 


0.00000 


1 


0.00000 


— 


MODE X CORRECT 


12.50000 


1 


12.50000 


1.136 


DIPPICULTY X 
MODE X CORRECT 


19.22000 


1 


19.22000 


1.747 


MODE WITHIN 


2111.83998 


192 


10.9917 




TOTAL 


2233.81998 


199 






* P <.05 













MEANS 

MEAN 


& STANDARD 

S.D. 


DEVIATIONS OP 


CELLS 

MEAN 


S.D. 


DCC 


2,28000. 


2.80654 


ECC 


4.00000 


3.62859 


DIG 


3.96000 


3.28481 


EIC 


4.36000 


3.23883 


DCNC 


4.04000 


2.86473 


ECNC 


4.52000 


3.60694 


DINC 


3.48000 


2.34734 


EINC 


5 . 12000 


4.35239 
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of difficulty and correction to be most often associated with 
the best performances. The question raised here Is, "Will the 
dimension of difficulty be associated with greater percentage 
of repeated errors from the Immediate test to the delayed 
test?" That Is, what percent of the time will a person who 
misses the Item on the Immediate test also miss the Item on 
the delayed test? Since the tests employed In this study are 
actually selected frames and part frames of the program, with 
equally difficult foils appended, one would expect that errors 
Induced by difficult choices on the program would tend to 
persist and be repeated on later tests. For easy choices. It 
Is noted that fewer errors were made on the program, but a 
greater number of errors were made on the first tests as com- 
pared to those who were exposed to difficult choices. One can 
only guess whether or not the same Items will be missed on 
an Identical test administered at a later date. 

The analysis and descriptive data for Item error repetition 
for both the completion test and the multiple-choice test are 
presented In Tables 7 and 8. None of the results presented 
Is significant, but some tendencies were found when the 
completion test Is employed as the criterion measure. It Is 
noted that a smaller percent of error repetition is associated 
with the difficult alternatives combined with constructed 
response and the correction condition (DCC). This finding, 
though not significant. Is counter to the Idea that difficult 
alternatives will result In persistence of a particular type 
of error. 
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TABLE 7 

ANALYSIS OP VARIANCE AND 
MEANS AND STANDARD DEVIATIONS USING 
ITEM ERROR REPETITION ON COMPLETION TESTS 
AS THE CRITERION MEASURE 



SOTmCFTSP 

VARIATION 


SUMS OP 

SQUARES 


df 


— RE5H 

SQUARES 


P 


DIFFICULTY 


0.22526 


1 


0.22526 


3.391 


RESPONSE MODE 


0.00000 


1 


0.00000 


— 


DIFFICULTY X 
MODE 


0.00445 


1 


0.00445 




CORRECTION 


0.02632 


1 


0.02632 


mmrnmmmmm 


DIFFICULTY X 
CORRECT 


0.04304 


1 


0.04304 




MODE X CORRECT 


0.15466 


1 


0,15466 


2.328 


DIFFICULTY X 
MODE X CORRECT 


0.20241 


1 


0.20241 


3.047 


MODE WITHIN 


12.75460 


192 


0.06643 




TOTAL 


13.41074 


199 









MEANS 

MEAN 


& STANDARD 
S.D. 


DEVIATIONS 


OP GELLS 
MEAN 


S.D. 


DCC 


0.59186 


0.35335 


EGG 


0.76138 


0.26837 


DIG 


0.72049 


0.22733 


EIC 


0.74389 


0.21042 


DCNC 


0.76338 


0.23431 


EGNG 


0.74697 


0.21889 


DING 


0.65353 


0.31206 


EING 


0.74551 


0.19599 





TABLE 8 



ANALYSIS OP VARIANCE AND 
MEANS AND STANDARD DEVIATIONS USING 
ITEM ERROR REPETITION ON MULTIPLE-CHOICE TESTS 





AS 


THE 


CRITERION 


MEASURE 




sotjrcK 0 ^" 
VARIATION 


"SUMS 15 F" 

SQUARES 




df 


MEAir ~ 

SQUARES 


P 


DIPPICUI/TY 


0.06836 




1 


0.06836 


— 


RESPONSE MODE 


0,01288 




1 


0.01288 


— 


DIPPICULTY X 
MODE 


0.70277 




1 


0.70277 


1.825 


CORRECTION 


0.14405 




1 


0.14405 




DIPPICULTY X 
CORRECT 


0.18311 




1 


0.18311 




MODE X CORRECT 


0.00110 




1 


0.0110 


— 


DIPPICULTY X 
MODE X CORRECT 


0.00527 




1 


0.00527 




MODE WITHIN 


73.91668 




192 


0.38498 


— 


TOTAL 


75.03422 




199 






MEANS & STANDARD DEVIATIONS OP CELLS 




MEAN 


n 

0 


• D. 




MEAN 


S.D. 



DCC 


0.23111 


1.00560 


ECC 


0.45743 ■ 


0.39501 


Die 


0.38067 


0 . 38645 


EIC 


0.34933 


0,40278 


DCNC 


0.36025 


0.39552 


ECNC 


0.44500 


0.36418 


DINC 


0.47990 


1.09367 


EINC 


0.34808 


0.34034 
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Percent of Response Errors 
Identically Repeated on Both Completion 
and Multiple-Choice Tests 

If S made a particular response error when either test 
was Immediately administered, did he also make that same response 
error on that same item when the teat was readmlnlstered at a 
later date? The data re^ported In Tables 9 and 10 are based on 
the percent of Identical response error repetition from the 
Immediate completion test to the delayed completion test and 
from the Immediate multiple-choice test to the delayed multiple- 
choice test. Only one significant effect Is reported, that of 
the difficulty dimension when the completion test Is employed 
as the criterion measure. Inspection of the means shows a 
smaller percent of Identical response error repetitions con- 
sistently associated with the difficult alternatives. That 
is, a greater proportion of Identical response error repetitions 
are associated with easy alternatives on the program rather than 
difficult alternatives. This finding Indirectly refutes the 
contention of Skinner (1958) and Porter (1957) that one tends 
to learn the errors made on the program and that these errors 
will persist, 

Pollow-up analyses show that a greater Incidence of 
identical response error repetition Is associated with the easy 
alternatives for all comparisons between difficult alternatives 
and easy alternatives, but none of these comparisons attained 
significance. Apparently, these consistent differences resulted 
In a significant; main effect for the difficulty dimension. 
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TABLE 9 

ANALYSIS OP VARIANCE AND 
MEANS AND STANDARD DEVIATIONS USING 
IDENTICAL RESPONSE ERROR REPETITION ON COMPLETION TESTS 

AS THE CRITERION MEASURE 



SOUr.CE C5P 
VARIATION 


SUMS OP 

SQUARES 


df 


— REan 

SQUARES 


p 


DIPPICULTSr 


0.22367 


1 


0.22367 


4.024 * 


RESPONSE MODE 


0.01783 


1 


0.01783 


— 


DIPPICULTY X 
MODE 


0.00052 


1 


0.00052 




CORRECTION 


0.01262 


1 


0.01262 


— — 


DIPPICULTY X 
CORRECT 


0.00671 


1 


0.00671 




MODE X CORRECT 


0.15653 


1 


0.15653 


2«8l6 


DIPPICULTY X 
MODE X CORRECT 


0.00753 


1 


0.00753 




MODE WITHIN 


10.67093 


192 


0. 05558 





TOTAL 


11.09634 


199 






* P < .05 





MEANS & STANDARD 
MEAN S.D. 


DEVIATIONS 


OP CELLS 

MEAN 


S.D. 


DCC 


0.30400 


0.26107 


ECC 


0.39153 


0.27171 


Die 


0.35012 


0.25242 


EIC 


0.41954 


0.23197 


DCNC 


0.36792 


0.25042 


ECNC 


0.40773 


0.20101 


DINC 


0.27760 


0.1 9772 


EINC 


0.34838 


0.20714 



o 
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TABLE 10 

ANALYSIS OF VARIANCE AND 
MEANS AND STANDARD DEVIATIONS USING 
IDENTICAL RESPONSE ERROR REPETITION ON MULTIPLE-CHOICE TESTS 

AS THE CRITERION MEASURE 



'sorocE OP 

VARIATION 


— StJUSUP 

SQUARES 


df 


— FEnr 

SQUARES 


P 




DIPPICULTY 


0,22599 


1 


0.22599 


1.62 




RESPONSE MODE 


0.01416 


1 


0.01416 


— 




DIPPICULTY X 












MODE 


0.40030 


1 


0.40030 


2.674 




CORRECTION 


0.20597- 


1 


0 . 20597 


1.479 




DIPPICULTY X 












CORRECT 


0 . 32940 


1 


0.32940 


2.365 




MODE X CORRECT 


0.16245 


1 


0.16245 


1.166 




DIPPICULTY X 












MOD>. X CORRECT 


0.03274 


1 


0.03274 


— 




MODE WITHIN 


26.74647 


192 


0.13930 






TOTAL 


28.11748 


199 











MEANS & STANDARD 
MEAN S.D. 


DEVIATIONS 


OP CELLS 
MEAN 


S.D. 


DCC 


0.05778 


0.21545 


ECC 


0.32124 


0.36114 


Die 


0.24667 


0.29214 


EIC 


0.28000 


0.35013 


DCNC 


0.28571 


0.37538 


ECNC 


0.3356? 


0.34797 


DINC 


0.30943 


0.62453 


EINC 


0.23160 


0.27861 
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Inspection of the means and standard deviations on Table 9 
shows a general decline In the percent of Identical response 
error repetition with the greatest reduction resulting from 
the DCC condition. However, It Is also noted that seven of 
eight standard deviations are considerably larger, which 
probably accounts for the findings of no significance for any 
of the main effects. 

Comparison of the Usual Linear 
Program with the Dlfflcult-Construct- 
Noncorrect Condition 

It was previously stated that the usual linear program 
(RCNC) was administered and the same criterion measures taken 
In order to provide a basis, for comparison with the other 
conditions of the study and to specifically compare these 
results with the results of the dlff Icult-construct-noncorrect 
(DCNC) condition. The results are presented In Table 11. 

The only significant result obtained by these comparisons was 
on the criterion measure of program errors. The DCNC condition 
resulted In significantly fewer program errors. Two other 
comparisons, that of time taken to complete the program 
and the percent of Identical response error repetitions, favored 
the DCNC condition. All others favored the RCNC condition. 

The Ss under the RCNC condition consistently performed better 
than the Ss under the DCNC condition, but not significantly so. 
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TABLE 11 



MEANS, VARIANCES, AND T-TESTS 
ON COMPARISONS OP THE RT^C PROGRAM 
WITH THE DCNC PROGRAM 



VAftlABLE 


PROGRAM 


— mm— 




“T-'tEST 


TOTAL PROGRAM ERRORS 


RCNC 


44.88 


406.10 


-2.26* 




DCNC 


31.12 


351.52 


NO. ERRORS IMMEDIATE 


RCNC 


7.65 


27.05 


-0.588 


COMPLETION TEST 


DCNC 


8.68 


33.72 


NO. ERRORS IMMEDIATE 


RCNC 


2.88 


5.87 


-0.045 


MULTIPIE-CHOICE TEST 


DCNC 


2.92 


9.49 


NO. ERRORS DELAYED 


RCNC 


10.35 


26.70 


- 1.42 


COMPLETION TEST 


DCNC 


12.56 


22.76 


NO. ERRORS DELAYED 


RCNC 


3.12 


4.46 




MULTIPLE -CHOICE TEST 


DCNC 


4.04 


8.21 


-1.13 


ITEM ERROR REPETITIONS 


RCNC 


.705 


.063 




ON COMPLETION TESTS 


DCNC 


.763 


.055 


- 0.733 


IDENTICAL RESPONSE 


RCNC 


.384 


.091 




ERROR REPETITIONS 


DCNC 


.367 


.063 


- 0.200 


ON COMPLETION TESTS 


ITEM ERROR REPETITIONS 


RCNC 


.241 


.113 




ON MULTIPLE-CHOICE TESTS 


DCNC 


.360 


.156 


- 1.01 


IDENTICAL RESPONSE 


RCNC 


.183 


.068 




ERROR REPETITIONS 


DCNC 


.286 


.141 


-0.990 


ON MULTIPLE-CHOICE TESTS 


TIME TAKEN TO 


RCNC 


105 . 12 


214.34 


-0.829 


COMPLETE PROGRAM 
(IN MINUTES) 


DCNC 


100.52 


378.76 



*p< .05 
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Additional Analyses 

In the course of conducting research the Investigator 
usually has some new "insights" or sees the project In a some- 
what different light than formerly. Such was the case in ‘ohls 
Investigation, On the following pages a brief discussion of 
the procedures and findings of selected analyses will be 
presented. 



A Comparison of Performance on 
Two Forms of Multiple-Choice Tests Under the 
Diff icult-Constructed Response-Noncorrectlon Condition 

The multiple-choice test used In this study was developed 
by selecting "critical" frames and part-frames from the program 
and appending difficult multiple-choice foils to those frames. 
However, these difficult foils were not the same as those on 
the modified program and therefore may have Interfered with the 
response of those Ss who were previously exposed to yet other 
difficult alternatives. Or would having the same identical foils 
on both the program and the test result In more errors? In order 
to clarify the issue a group of Ss numbering 25 were administered 
the difficult alternative-constructed response-noncorrectlon (DCNC) 
program. It Is recalled that one other group responded to this 
same program. The majf r difference between this group and the 
group who earlier took this program was that this group was 
administered both the old completion and multiple-choice tests 
with the difficult alternatives of the program appended to each 
frame. Now If an ^ makes an error on the program, will that 
error tend to persist through both the Immediate and delayed 
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tests because the cues are Identical from the learning to the 
test situation? Will the error rate and error repetition be 
greater for the DCNC condition which takes the test with the 
identical foils or will Ss who take the test with different 
difficult foils be more prone to error production and error . 
repetition? T^ta bearing on these questions is presented in 
Table 12. Within this table the original group is labeled DCNC, 
and the group with the tests which have foils identical to those 
on the learning program are labeled DCNC-Identical Foils Test 
(IPT). The comparisons of interest are those associated with 
the multiple-choice tests as opposed to the completion tests. 

In every comparison the group which took the multiple-choice 
test with the same foils as appeared on the program either 
made more errors than the original DCNC group or made signifi- 
cantly more error repetitions. Apparently, re-presenting the 
identical cue situation as was encountered on the program results 
in greater error production and error repetition as compared to 
a test situation in which the alternatives to the correct re- 
sponse were difficult but different. 

Analysis of Program Errors 

A number of errors made on the learning program could be 
repeated on the tests that followed since the tests consisted 
of a sampling of the frames of the program. However, the number 
of possible error repetitions per S was relatively small. The 
number of identical response error repetitions, item error repe- 
titions, and item corrections from the program to the test are 
reported in Tables 13 and 14 for the program conditions of RCNC, 
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TABLE 12 

MEANS, VARIANCES, AND T-TESTS 
ON COMPARISONS OP THE DCNC-IFT GROUP 
WITH THE ORIGINAL DCNC GROUP 



VARIABLE 


TRUUKAH'-^l'JOT 


MEaiIS 


VARIANCES 


T-TEST 


TOTAL PROGRAM ERRORS 


DCNC-IPT 

DCNC 


34.08 

31.12 


407.11 

351.52 


-0.531 

-0.531 


NO. ERRORS IMMEDIATE 
COMPLETION TEST 
(DCNC-IFT had multiple- 
choice here) 


DCNC-IPT (M-C) 
DCNC (Comp.) 


5.04 

8.68 


12.99 

33.72 


-2.66* 


NO. ERRORS imnEDlATE 
MULTIPLE-CHOICE TEST 


DCNC-IFT 

DCNC 


4.84 

2.92 


9.89 

9.49 


-2.18* 


NO, ERRORS DELAYED 
COMPLETION TEST 
( DCNC-IFT had multiple- 
choice here) 


DCNC-IPT (M-C) 
DCNC (Comp.) 


5.60 

12.56 


12.24 

22.76 


-5.88** 


NO. ERRORS DELAYED 
MULTIPLE-CHOICE TEST 


DCNC-IPT 

DCNC 


6.72 

4.04 


10.68 

8.21 


-3.08** 


PERCENTAGE OF ITEM 
ERROR REPETITIONS ON 
MULTIPLE-CHOICE TESTS 


DCNC-IPT 

DCNC 


.652 

.361 


.065 

.156 


-3.09** 


PERCENTAGE OF IDENTICAL 
RESPONSE ERROR ON 
MULTIPLE-CHOICE TESTS 


DCNC-IPT 

DCNC 


.524 

.286 


.074 

.141 


-2.57* 


TIME TAKEN TO COMPLETE 
PROGRAM (IN MINUfTES) 


DCNC-IPT 

DCNC 


105 . 88 
100.52 


937.63 

378.76 


-0.738 



*p< .05 

**p< .01 
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TABLE 13 

A COMPARISON OP THE DCNC-IPT AND RECALL- CONSTRUCT 
PROGRAMS ON PROGRAM ERRORS MISSED OR CORRECTED 
ON IMMEDIATE AND DELAYED CRITERION TESTS 



IMMEDIATE TEST DCNC PROGRAM WITH 

IDENTICAL MULTIPLE- 
CHOICE FOILS TEST 



RECALL CONSTRUCT 
PROGRAM COMPLETION 
TEST 





Mean 


% of 
Total 


Mean 


56 of 
Total 


POSSIBIiE ftEPEATABlE’ 
PROGRAM ERRORS 


5.454 




7.118 ^ 




TEST ITEMS MISSED 
IDENTICAL ERROR 


1*545 


38.335^ 


1.118 


15.71^ 


TEST ITEMS MISSED 
DIFFERENT ERROR 


0.591 


10.84$^ 


2.118 


29.76^ 


TEST ITEM RIGHT 
ERROR ELIMINATED 


3.318 


60.8356 


3.882 


54.5356 


DELAYED TEST 










POSSIBLE REPEATABEET 
PROGRAM ERRORS 5.^54 




7.118 




TEST ITEMS MISSED 
IDENTICAL ERROR 


1.136 


20.8356 


1.118 


15.71^ 


TEST ITEMS MISSED 
DIFFERENT ERROR 


0.864 


15 . 8456 


2.647 


37.1956 


TEST ITEM RIGHT 
ERROR ELIMINATED 


3.454 


63.33^ 


3.353 


47.1056 



Note: Comparison may not be entirely valid since the only 
common denominator Is the lead statement In the test 
Items. The DCNC-IPT group took the DCNC multiple- 
choice program while the other group took the usual 
linear program. The DCNC-IPT group then took the 
Identical foils multiple-choice format on these items 
while the RCNC group responded to a completion test. 



TABLE 14 



A COMPARISON OP THE DCNC-IPT AND DCNC PROGRAMS 
ON PROGRAM ERRORS MISSED OR CORRECTED 
ON IMMEDIATE AND DELAYED CRITERION TESTS 



IMMEDIATE TEST 



DCNC PROGRAM WITH 
IDENTICAL FOILS 
MULTIPLE-CHOICE 
TEST 

% of 



DCNC PROGRAM WITH 
DIFFERENT FOILS 
MULTIPLE-CHOICE 
TEST 

% of 



POSSIBLE IfepfeATABLE 
PROGP-AM ERRORS 


5.8^6 




4.64 




TEST ITEMS MISSED 
IDENTICAL ERROR 


1.652 


28.36jg 






TEST ITEMS MISSED ’ 
DIFFERENT ERROR 


0.783 


13.44^ 


0.92 


19 . 835 ? 


TEST ITEMS RIGHT 
ERROR ELIMINATED 


3.391 


59 . 20 ^ 


3.72 


80 . 175 ? 


DELAYED TEST 










rUooIl^JLill UtLJr'iiiATABLE 
PROGRAM ERRORS 


5.826 




4.64 




TEST ITEMS MISSED 
IDENTICAL ERROR 


1.608 


27. 6C^ 


^ 




TEST ITEMS MISSED 
DIFFERENT ERROR 


0.957 


16.43^ 


1.32 


28.455? 


TEST ITEMS RIGHT 
ERROR ELIMINATED 


3.261 


55 . 91 % 


3.32 


71.55^ 



these categories for identical error 

employed different foils 

from those used on the corresponding program frames 



ERIC 
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DCNC, and DCNC-IPT. The findings of Interest are that the DCNC 
condition resulted in the largest percentage of program error 
corrections from the program to the test and the RCNC (linear) 
condition the smallest percentage. That Is, program errors tend 
to persist more often under the RCNC condition with the completion 
test as the criterion measure as compared to the other two condi- 
tions with a multiple-choice test as the criterion measure. The 
small number of cases on which this data is based means that this 
data Is suggestive only. 

In addition, the DCNC-IPT condition resulted In a smaller 
percentage of program error corrections than did the DCNC condi- 
tion. The use of the same foils on both the program and the 
multiple-choice criterion test (the DCNC-IPT condition) tends 
to Interfere with the process of error correction. 

Efficiency Measures 

The question of the efficiency of program conditions has 
been raised on occasion. It was expected that the condition 
which included the easy alternatives and the indicated response 
would require less total program time, which Is the case, and 
unless the program conditions are greatly Ineffective In regard 
to learning one would expect the above conditions to be associ- 
ated with more favorable efficiency Indices. 

Time taken to complete the program, mean number right, 
minutes on the program needed to produce one correct response, 
number of Items learned per minute on the program, and rank order 
of the program conditions on the basis of the latter measure 
are presented In Tables 15 and 16. It Is noted that the EIC 
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TABLE 15 

MEAN PROGRAM TIME, MEAN NUMBER CORRECT 
AND RESULTING EFFICIENCY INDICES BASED ON 
IMMEDIATE AND DELAYED COMPLETION TESTS 



Type of 
Program 


program 

Time 

(Minutes) 


Mean Mumper 
of Test 
Items Correct 


Minutes to 
Produce One 
Correct Response 


Item Mastery 
Rate Per 
Minute 


Rank 

Order 


Immediate Completion 


Test 








DCC 


95.56 


22.80 


4.19 


.239 


4 


DCNC 


100.52 


19.32 


5.20 


.192 


9 


Die 


82.76 


29.68 


4.00 


.249 


3 


DING 


88.80 


18. 40 


4.83 


.207 


7 


ECC 


83.64 


19.08 


4.38 


.223 


6 


ECNC 


77.48 


18.20 


4.25 


.235 


5 


EIC 


71.12 


18.44 


3.85 


.259 


1 


EINC- 


69.32 


17.52 


3.96 


• 253 


2 


RCNC 


105 . 18 


20.35 


5.17 


.193 


8 


Delayed 


Completion Test 








DCC 


95.56 


18.00 


5.30 


.188 


5 


DCNC 


100.52 


15.44 


6.51 


.154 


9 


Die 


82.76 


16.24 


5.10 


.196 


4 


DINC 


88.80 


15.08 


5.89 


.169 


7 


ECC 


83.64 


14.88 


5.62 


.178 


6 


ECNC 


77.48 


15.40 


5.03 


.199 


3 


EIC 


71.12 


16.54 


4.30 


.232 


1 


EINC 


69.32 


15.00 


4.62 


.216 


2 


RCNC 


105 . 18 


17.65 


5.96 


.168 


8 



1 
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TABLE 16 

MEAN PROGRAM TIME, MEAN NUMBER CORRECT, 
AND RESULTING EFFICIENCY INDICES BASED ON 
IMMEDIATE AND DELAYED MULTIPLE-CHOICE TESTS 



Type of 
Program 


Program 

Time 

(Minutes) 


Mean Number 
of Test 
Items Correct 


Minutes to 
Produce One 
Correct Response 


Item Mastery 
Rate Per 
Minute 


Rank 

Order 


Immediate 


Multiple-Choice Test 








DCC 


95.56 


26.72 


3.57 


.280 


7 


DCNC 


100.52 


25.08 


4.01 


.250 


8 


Die 


82.76 


25.64 


3.23 


.310 


4 


DING 


88.80 


25.24 


3.56 


.284 


6 


ECC 


83.64 


24.96 


3.35 


.298 


5 


ECNC 


77.48 


25.16 


3.08 


.325 


3 


ETC 


71.12 


25.76 


2.76 


.362 


1 


EINC 


69.32 


24.72 


2.80 


.357 


2 


RGNC 


105 . 18 


25.12 


4.18 


.239 


9 



Delayed Multiple-Choice Test 




I5CC 


95.56 


25.72 


DCNC 


100.52 


23.96 


Die 


82.76 


24.04 


DING 


88.80 


24.52 


see 


83.64 


24.00 


ECNC 


77.48 


23.48 


EIC 


71.12 


23.64 


EINC 


69.32 


22.88 


RCNC 


105.18 


24.88 



3.72 


.269 


7 


4.20 


.238 


8 


3.44 


.290 


4 


3.62 


CVJ 

• 


6 


3.49 


.287 


5 


3.20 


..303 


3 


3.01 


.332 


1 


3.03 


.330 


2 


4.23 


.237 


9 
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and EINC are most efficient In every case with the DC and RC 
group generally showing as least efficient. The major reason 
for these results is simply that the latter conditions require 
considerably more time In going through the program, but cannot 
perform that much better on the criterion tests. The efficiency 
index, or at least some record of time taken to complete the 
program. Is useful Information If two groups should happen to 
score equally well on the criterion tests. Then the most effi- 
cient program would be desirable. 






o 
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Discussion 



Program Errors 

^ As expected, appending the difficult and easy alternatives 

; to the linear program frames resulted in great discrepancies in 

if 

program error rates with fewer errors being associated with the 
easy alternatives. More Interesting was the fact that (a) con- 
structed responses, as opposed to indicated responses, and 
(b) the error correction procedure, in comparison to the usual 

t 

non-correction procedure, both yielded fewer program errors. 

As was previously mentioned, comparisons between indicated and 
constructed responses are lacking except in those cases in 
which the Indicated response is confounded with the multiple- 
choice frame and the constructed response is confounded with 
the completion frame. But in this comparison in which both 
groups had to select a response from among a number of alter- 
native responses and then either write or Indicate the response 
selected, writing the response resulted in fewer program errors. 
This result was not predicted, but one possible explanation of 
it is that writing the response allowed more time for covert 
rehearsal of the response. Thus the response was better learned 
and not so easily missed in the later portions of the program. 
Perhaps Underwood’s (1959) discussion of the relationship 
between frequency and availability of a response may be appro- 
priately cited as an explanation of the effects of rehearsal. 

Finding that the correction procedure resulted in fewer 

] program errors was, in part, expected. Some support for the 

i 

I effectiveness of a procedure which requires making a correct 

1 

•! 

i 








response last in a response sequence would be found in Guthrie's 
recency principle. As frames are encountered similar to the 
one in which S missed the response, and corrected, one would 
expect fewer errors to be made by the Ss making the correction 
on the earlier frames. 

The last significant effect to be reported using program 
errors as the criterion measure is a two-way interaction between 
difficulty and correction. This result is explained by noting 
that the difficult alternatives resulted in a greater number 
of program errors thus providing an optimum condition for the 
correction procedure to be effective. It appears that such 
overt correction procedures will allow for additional rehearsal 
of the correct response. 

Tests as Criterion Measures 

None of the studies previously cited found the multiple- 
choice response to be superior to the constructed response 
using tests as the criterion measures. (Burton and Goldbeck, 
1962; Coulson and Sllberman, I960; Pry, 196O; Hough, 1962; 

Moore and Smith, 1964; Williams, 1965). The findings of this 
study, in which the response mode was not separate from the 
multiple-choice response, did not show the constructed response 
to be the significant factor when the criterion measure was 
the immediate completion test constructed of selected frames 
and part-frames of the program. It was predicted that those 
^s required to construct a response would make fewer errors. 

But the factors of difficulty and correction were both statis- 
tically significant, the former at the .01 level of confidence 
which was not predicted. It should also be recalled that the 




RCNC (linear) condition resulted In a larger percent of errors 
than either of the conditions of difficult alternatives with 
correction procedure on this criterion measure. Use of the 
difficult alternatives with the correction procedure may have 
resulted in greater scrutiny of the stimulus statements of 
each frame and thus in better performance on the criterion 
test. It is also possible that the correct response was given 
somewhat more rehearsal time under these conditions and there- 
fore was better learned. 

However, the completion test administered after an 
approximate two-week delay failed to yield significant differ- 
ences for the same two factors. The RCNC (linear) condition 
was found to be almost as effective as the most effective 
condition (DCC) under analysis. Apparently, the advantage 
accorded the dimensions of difficulty and correction is tempo- 
rary. In addition, one can only speculate on the effectiveness 
of the usual linear program with the added condition of the 
correction procedure. 

The results were exactly the reverse when the multiple- 
choice test was administered both immediately and approximately 
two v/eeks later. None of the factors under study in this 
investigation showed significant results when the test was 
administered Immediately, but under delayed administration 
of the same test the dimension of difficulty was significant. 
However, the factor of difficulty approached significance at 
the ,05 level on the immediate administration of the multiple- 
choice test. Again it is noted (Table 1 ) that the RCNC condition 



showed less loss over time than any of the experimental condi- 
tions of this study. 

Summary of Data Using Program 
Errors and Tests as Criterion Measures 

Those conditions (the easy alternatives) which were 
associated with the fewest program errors did not result in 
superior test performance as compared to those conditions 
associated with a greater program error rate. However, among 
the conditions associated with the difficult alternatives the 
condition which resulted in the least number of program errors 
also resulted in superior test performance. But the condition 
which resulted in the greatest number of program errors was 
the usual linear program which has been designated RCNC, In 
the light of this evidence the arguments of Skinner (1958) 
and Porter (1957) j that multiple-choice foils would result in 
error production which in turn would Interfere with the 
learning process, do not seem Justified, Instead it would 
appear that the difficult program alternatives tend to pro- 
mote inspection behavior of the stimulus section of the frame 
and perhaps some rehearsal behavior of the response term as well. 

Only the difficulty dimension was effective in reducing 
the number of errors made on the criterion tests of the nature 
used in this study. The correction procedure was effective in 
some of those instances when it was used with difficult alter- 
natives, Although the response mode was not a significant fac- 
tor it is noted that the most effective condition throughout 
was the DCC condition with the DIG and RCNC conditions next 
In effectiveness. 
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Error Analysis 

The data on item error repetition and identical response 
error repetition over the two-week period on both test types 
are not particularly convincing. Only one factor was statis- 
tically significant, that of difficulty when the criterion 
measure of Identical response error repetitions was employed. 
Further scrutiny of the data showed that the tendency to 
repeat response errors was associated with the easy alterna- 
tives and not the difficult ones as might be expected. This 
tendency for the test error repetitions to be associated 
with the easy alternatives may be noted by inspecting the 
means of the tables in which results were presented, 

A convincing explanation for this phenomena is dif- 
ficult to achieve. It seemed reasonable that the difficult 
alternatives would tend to result in more program errors 
(which happened) and thus the tendency to repeat or fixate 
on these errors should be noted even when test-to-test 
errors were measured. One problem which complicates the inter- 
pretation is that on the completion test there are no alterna- 
tives while on the multiple-choice test the alternatvies were 
in every case different from those employed on the program. 

How will the program errors affect test-to-test errors under 
these conditions? 

The most logical explanation Is that Ss exposed to the 
difficult alternatives. Instead of fixating their responses, 
actually have more legitimate alternatives to choose from 
and on this basis, partly by guessing, vary their response from 
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test to test, and perhaps from program to test as well, though 
this was not ascertained. Those ^s exposed to easy alternatives 
had been exposed to fewer legitimate alternatives and were not 
as able to vary their response. The same general statement 
may be made for the RCNC condition for which a response error 
repetition percentage was more like that of the ^s exposed to 
easy alternatives ( 38 . 4 ^). Exposure to difficult and legiti- 
mate alternatives may not result In more fixed behavior, but 
In fact results In more flexible behavior. This result was 
not found at the expense of a poorer, overall test performance 
of those ^s exposed to difficult alternatives. It appears 
that something else may be learned when plausible alternatives 
were added to the frames which may result In superior transfer 
behavior as the knowledge Is used or applied because of the 
availability of more Information about likely and unlikely 
responses to a stimulus statement. 

Those subjects exposed to easy foils were not required 
to make discriminations among plausible responses on the pro- 
gram, but were exposed to plausible response alternatives on 
the criterion test. If they did not know the correct response, 
then they tended to make the wrong response on some basis of 
logic that seemed appropriate to them and which process tended 
to be repeated each time they encountered that particular Item. 

The subjects exposed to the difficult multiple-choice alter- 
natives presumably learned to discriminate among plausible 
alternatives. If these subjects responded Incorrectly they 
tended to more often vary their response when they next encountered 
the Item because they had been exposed to plausible alternatives 
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before and associated a number of the plausible alternatives 
with that stimulus statement. It would seem that on the basis 
of this discussion one could state that those ^s exposed to 
difficult multiple -choice alternatives had learned more than 
those exposed to easy alternatives. Whether this Is "good" 
or "bad" Is still debatable. 

Other Analyses 

The comparison between the DCNC and RCNC program was 
made to evaluate the effect of only the difficult alternatives 
on Ss performance. The results Indicated little difference 
In the performance of either group exposed to these two pro- 
gram formats except on the criterion of program errors, the 
greater number of errors being associated with the RCNC pro- 
gram. Appending the difficult alternatives did not add to the 
effectiveness of the program when tests were the criterion 
measures. These results tend to agree with a number of other 
studies previously cited in which no differences were found 
between multiple-choice formats and the usual linear programs. 
Little transfer was noted from the program response format to 
the test response format with Ss exposed to the DCNC and RCNC 
programs performing equally well on either type of test. 

The DCNC and DCNC-IPT comparisons showed significantly 
more errors and error repetitions being associated with the 
DCNC-IPT condition on the criterion tests. It is recalled 
that some of the comparisons were not meaningful as the DCNC 
group responded to a completion test and the DCNC-IPT group 
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responded to the same test except that the same alternatives 
used on the program were appended to the completion statements. 

It appears that re-present Ing the "Identical" cue situ- 
ation on the test as It was encountered In the program results 
In poorer performance. Encountering the same difficult alter- 
natives on both the program and the test appears to extend 
the effects of the difficulties In making a choice on the pro- 
gram. That Is, the foils encountered on both the program and 
the test are more familiar because of the prior exposure and 
It therefore becj^mes a more difficult task In selecting one 
of these familiar foils as being correct. Encountering a dif- 
ferent set of foils, except for the one which Is correct, on 
the criterion test would then pose a somewhat easier task. 

The ^ may more easily recognize the correct response as being 
correct when It Is placed with unfamiliar foils. 

A possible explanation of these results is the von Res- 
torff phenomenon (Wallace, 1965) on the superior recall of 
materials which are in some manner Isolated from other materi- 
als In which the first set of materials Is embedded. For 
example, if one pair of a list of paired associated is of 
heterogeneous material and the remaining pairs are of homo- 
geneous material, the heterogeneous pair will be better 
retained than the remaining pairs. In a sense, prior expo- 
sure to one term among four alternatives may tend to Isolate 
that term from the others thus aiding in the recall of the 
correct response. Then encountering a different set of alter- 
natives, except for the correct response, would aid In select?- 
Ing the correct response. 
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It would be of Interest to know If Ss responding to the 
same Identical foils on both the program and the test tend to 
repeat the Item errors made on the program to a greater degree 
than the DCNC group which took a test composed of different 
difficult foils. Possibly the reappearance of the response 
they originally chose and missed leads them to select that 
same response once again, whereas the wrong response Is not 
available fo. the DCNC group so that their choice shifts to 
the most familiar response which happens to be the correct 
one, A different research attack would be needed to answer 
the question of why these results obtained. 

The analysis of program errors suggests that repetition 
of program errors on the test for the DCNC-IPT condition may 
have accounted for the results Just discussed. That is, the 
DCNC group had a larger percentage of error corrections from 
program to test than did the DCNC-IPT group. An Incidental 
finding which was earlier reported was that the RCNC condition 
resulted In the smallest percentage of error correction of 
Items missed on the program and the largest oercentage of test 
ittms missed with an error different than that which was made 
on the program. Program errors tend to endure under the DCNC- 
IPT and RCNC conditions. These results are only tendencies 
as the data on which the analysis Is based are quite meager, 
Plnally, the matter of efficiency is of concern only 
If different conditions result In equally good performances, 
but some conditions require less time in learning the mate- 
rials, The Difficult -Construct-Correct condition was 
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superior to all other conditions on performance and required 
less time than two of the program conditions including the 
usual linear program. 

Conclusions and Implications 
Conclusions 

The major conclusions to he drawn from this study ares 

I. Easy alternatives, constructed responses, and error cor- 
rection resulted in a reduced error rate on the program,, 
but the error correction procedure was effective in 
reducing errors only when difficult alternatives were 
employed. The error rate for the usual linear program 
was higher than when difficult alternatives were appended. 
This result is in contrast to the belief that using plaus- 
ible multiple-choice alternatives with a linear program 
will tend to Increase the error rate. 

II. Difficult foils and the error correction procedure 

resulted in superior performance on the completion test 
administered immediately . None of the factors in this 
study resulted in differential performance on the delayed 
completion test. The usual linear program was more effective 
than any of the conditions except those in which difficult 
foils and the correction procedure were employed. 

III. Only the difficult alternatives affected performance on 

the multiple-choice test and then only on the delayed test. 
Only the condition in which difficult foils, constructed 
response, and the error correction procedure were used was 
superior to the usual linear program on this criterion 
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measure. 
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IV, Easy alternatives and the usual linear program promoted 
Identical response error repetitions from the Immediate 
to the delayed test. This result wa's explained by sug- 
gesting that the Ss under these conditions did not have 
as many plausible alternatives to choose from as did those 
Ss v/ho were exposed to the difficult alternatives thus 
tended to repeat their response on both tests, 

V, The usual linear program was as effective on a variety 
of criteria as a modified program to which difficult 
foils were appended and Ss were required to construct 
the responses. 

VI. A dlff Icult-construct-noncorrect program given to two 
groups, one which later responded to a test with alter- 
natives Identical to those on the program and the other 
to tests which had difficult foils which were different 
from those on the program resulted In superior test 
performance for the latter group. These findings were 
discussed with perhaps the best explanation being that 
familiarity with the same set of plausible responses 
from program to test resulted In more confusion In 
selecting the correct response than when the correct 
response Is presented with a new set of difficult alter- 
natives. 

VII. The efficiency Index, a ratio of time spent on the pro- 
gram to correct responses made on the tests Is useful 
only If test performance Is approximately equal, and less 
time Is required for one program as compared to the others. 
Under these conditions the difficult-correct modifications. 
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regardless of mode of response, were the most efficient 
programs, being somewhat more effective than the usual 
linear program. 

Implications 

The results of this study suggest that the use of dif- 
ficult, plausible alternatives coupled with a correction 
procedure may be the most effective adaptation of the linear 
program. Those ^s who took this program out-performed those 
exposed to the usual linear program on every criterion measure 
employed. Additional research using the error correction 
procedure with the typical linear program is needed and would 
be of Interest, 

Further study of the effect of these various conditions, 
plus that of the usual linear program, on the repetition of 
item and response errors. Is essential. There was insufficient 
data available to study the direct effect of program errors 
on test errors. The investigator diverted his attention from 
these data to test-to-test comparisons which are of less 
Interest. Those data which were available on the effect of 
program errors on test performance were provocative and 
suggest that plausible alternatives serve a useful function 
in providing more effective discriminations between the 
correct response and the Incorrect alternatives. 

The criterion tests employed in this study were com- 
posed of frames and part-frames of the program. This type 
of test was employed to assess the direct effects of the 
various conditions of this study, A study of considerable 
Interest could be designed by developing criterion tests to 
test the subject's ability to transfer his learning under the 
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several program conditions employed In this study with the pcs 
sible exclusion of the easy alternatives. It was earlier 
suggested that those subjects exposed to the difficult alter- 
natives learned something different from those subjects 
exposed to easy alternatives on the usual linear program. 
Those subjects tended to show more flexible behavior; at 
least they were less inclined to repeat errors from the imme- 
diate to the delayed administration. The question raised 
here is, "Is this kind of behavior beneficial or detrimental 
to performance on test situations requiring transfer of 
learning?" It appears that some interesting questions in 
regard to program format remained unanswered by the research 
results presently available. 
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Appendix A 

An example of the learning program with the d ifficult 
multiple-choice alternatives appended. The itemsliere dis 
played are the first ten items of Set One of the Holland- 
Sklnner program, entitled The Analysis of Behavior . 



/ 



/ 




SET ONE 68 



1 


A doctor taps your knee (patellar tendon) with a rubber hasnmer to test 
your (A. RECOIL, B. RESPONSES, C. REACTIONS, D. REFLEXES), 


Correct 

— here 

<- — 




D. REFLEXES 


J 


If your reflexes are normal, your leg (A. JERKS, B, RESPONDS, C. REACTS, 
D, RECOILS) with a slight kick to the tap on the leg (the so-called 
knee-jerk) , 


Correct 




here 
<- — 


B. RESPONDS 



In the knee-jerk or patellar-tendon reflex, the kick of the leg Is the 
(A. MOVEMENT, B. RECOIL, C. RESPONSE, D. REFLEX) to the tap on the knee. 



Correct 

— here 

<- — 

C. RESPONSE 




4 


The tap delivered by the so-called stimulus object or hammer Is the 
(A. BESt>ONSE, B. STIMULUS, C. REACTION, D. REPLEX) which elicits the 
knee-jerk. 


Correct 

here 

<- — 

B. STIMULUS 




5 


The hammer used by the doctor to el: 
STIMl/LtJS, B. RESPONSE OBJECT, C. ST! 


icit a knee-jerk is called a (A. OBJECT 
IMULUS OBJECT, D. STIMULUS RESPONSE). 



Correct 

here 

<- — 



C. STIMULUS OBJECT 
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i 6 


In the knee-jerk reflex, we called the rubber hammer the (A. REFLEX 




OBJECT, B. ELICITER OBJECT, C. STIMULUS OBJECT, D. RESPONSE OBJECT) and 
the tap or blow, the (A. CAUSE, B. STIMULUS, C. REFLEX, D. RESPONSE). 


Correct 




here 


_ ^ 

C. STIMULUS OBJECT, B. STIMULUS 


7 


An event Is explained when Its cause Is Identified. The "cause" or 




explanation of the knee-jerk Is, technically, the (A. STIMULUS, 
B. RESPONSE, C. REFLEX, D. OBJECT) which elicits it. 


Correct 




here 

<- 


A. STIMULUS 


8 


Technically speaking a reflex involves an eliciting stimulus in a process 
called elicitation, A stimulus (A. STIMULATES, B. CAUSES, C. TRIGGERS, 

D. ELICITS) a response. 




Correct 




<- 


D. ELICITS 


9 


To avoid unwanted nuances of meaning in popular words, we do not say 
that a stimulus triggers, stimulates, or causes a response, but that it 
(A. CAUSES, B. ELICITS, C. STIMULATES, D. TRIGGERS) a response. 


Correct 

hpr»P 


• 


<- — 


B. ELICITS 


10 


In a reflex, the stimulus and the elicited response occur in a given 
temporal order, first the (A. STIMULUS, B. EFFECT, C. CAUSE, D. RESPONSE) 
and then the (A. CAUSE, B. STIMULUS, C. EFFECT, D. RESPONSE). 


Correct 

here 

^ <- — 

1 A. STIMULUS, D, RESPONSE 


• 



Appendix B 



An example of the learning program with the easy multiple 
choice alternatives appended. The items here displayed are 
the first ten items of Set One of the Holland-Skinner program, 
entitled The Analysis of Behavior. 
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SET ONE 



1 



II 

A doctor taps your knee (patellar tendon) with a rubber hammer to test 
your (A. SOURCES, E, CUES, C. CAUSES, D. REFLEXES). 



* 


Correct 




^ D. 


<- — 

REFLEXES 


2 


If your reflexes are normal, your leg (A, CONCLUDES, B, RESPONDS, ' 




C. ENDS, D, FINISHES) with a slight kick to the tap on the leg 




(the so-called knee-jerk). 







Correct 




B. 


<. 

RESPONDS 




3 


In the knee-jerk or patellar-tendon reflex, the kick of the leg Is 

the (A. BEGINNING, B. CUE, C. RESPONSE, D. CAUSE) to the tap on 
the knee. 





Correct 
here 
<- 



1 



C. RESPONSE 



The tap delivered by the so-called stimulus object or hammer Is the 
(A. OUTCOME, B. STIMULUS, C. RESULT, D. EFFECT) which elicits the 
knee-jerk. 



Correct 

here 



B. STir^LUS 



f 




The hammer used by the doctor to elicit 
(A. RESULT OBJECT, B. OUTCOME OBJECT. C. 
OBJECT). 



knee-jerk Is called a 
STIMULUS OBJECT, D. EFFECT 



STIMULUS OBJECT 










Correct 

here 
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I SET ONE 72 

6l In the knee»Jerk reflex, we called the rubber hammer the (A. PRODUCT 

* OBJECT, B. RESimr OBJECT, C. STIMULUS OBJECT, D. CONCLUSION OBJECT) 

and the tap or blow, the (A. RESULT, B. STIMULUS, C. EFFECT, D. OUT- 
COME). 



Correct 

here 

<- 



C. STIMULUS OBJECT, B. STIMULUS 



71 An event Is explained when Its cause Is identified. The "cause" or 
— ® explanation of the knee-jerk Is, technically, the (A. STIMULUS, 

B, PRODUCT, C. RESULT, D. CONCLUSION) which elicits It. 



Correct 

here 

<- 



A. 



8 



STIMULUS 



Technically speaking a reflex Involves an eliciting stimulus In a 
process called elicitation. A stimulus (A, vONCLUDES, B. ENDS, 

0. FINISHES, D. ELICITS) a response. 



Correct 

here 




D. 

9' 



ELICITS 



To avoid unwanted nuances of meaning In popular words, we do not say 
that a stimulus triggers, stimulates, or causes a response, but that 
it (A. CONCLUDES, B. ELICITS, C. ENDS, D. FINISHES) a response. 



B, 


Correct 

here 

<- 

. ELICITS 






1 In a reflex, the stimulus and the elicited response occur In a given 
temporal order, first the (A. STIMULUS, B. FINISH, C. OUTCOME, 

D. RESULT) and then the (A. START, B. CAUSE, C. CUE, D. RESPONSE). 



Correct 

here 

<— - 



A. STIMULUS. D. RESPONSE 
o 
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Appendix C 

The criterion test, including both multiple-choice 
and completion sections. 
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INSTRUCTIONS: Circle the LETTER of the correct answer 



2 . 



3 . 



4 . 



5 . 



6 . 



When testing for the presence of a conditioned reflex. It 



presented on the test triala, 

A. UNCONDITIONED RESPONSE 

B. NEUTRAL STIMULUS 

C. NEUTRAL RESPONSE 

D. UNCONDITIONED STIMULUS 

An electrically operated food magazine which presents food 
can be used to reinforce a(n) of a hungry 

organism. 

A* ELICITATION 

B. THRESHOLD 

C. LATENCY 

D. RESPONSE 

When a peck is followed by food, the event Is described by 
saying, "The peck was followed by . " 

A. REWARD 

B. REINFORCEMENT 

C. EXTINCTION 

D. CONDITIONING 

The mother can reinforce an Infant's vocal behavior only 
after at least one vocalization has been 



A, CONDITIONED 

B, EMITTED 

C, STIMULATED 
De EXTINGUISHED 

A process by which a stimulus loses the power to elicit a 
response is called . 

A. LATENCY 

B. EXTINCTION 

C. FORGETTING 

D. PAIRING 

The more Intense the stimulus, the shorter the 

of the reflex, 

A, THRESHOLD 

B. RESPONSE 

C. LATENCY 

D, LENGTH 



Is essential that the 



not be 
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7. In the knee-jerk reflex, the kick of the leg Is the 

to the tap on the knee. 

A. MOVEMENT 

B. ACTIVITY 

C. REACTION 

D. RESPONSE 

8. A response and Its eliciting stlmulns comprise a(n) 

A. BEPEEX 

B. ELICrrATION 

C. REACTION 

D. MOVEMENT 

9. The magnitude of a response Is a function of the 
of the stimulus which elicits It. 

A. RESPONSE 

B. VELOCITY 

C. INTENSITY 

D. THRESHOLD 

10. In 5 the word "candy" Is presented repeatedly 

without the unconditioned stimulus. 

A. ELICITATION 

B. CONDITIONING 

C. EXTINCTION 

D. REFLEXES 

11, An Important aspect of the conditioning procedure Is the 

, between the presentation of the neutral stim- 
ulus and of the unconditioned stimulus. 



A, REFLEX 

B, TIME 

C, THRESHOLD 

D, RESPONSE 

12. Reinforcement and behavior 

1 . 2 . 

1, A, EXTINCTION 

B, BEHAVIOR 

C, REINFORCEMENT 

D, CONDITIONING 



occur in the temporal order 



2. A. BEHAVIOR 

B, EXTINCTION 

C, CONDITIONING 

D, REINFORCEMENT 
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13* A response elicited by an unconditioned stimulus Is a(n) 



A. CONDITIONED RESPONSE 

B. UNCONDITIONED RESPONSE 

C. NEUTRAL STIMULUS 

D. UNCONDITIONED REFLEX 

l4. When a response Is elicited by a stimulus without previous 
conditioning, the sequence Is called a(n) 



A. UNCONDITIONED REFLEX 

B. CONDITIONED RESPONSE 

C. CONDITIONED REFLEX 

D. CONDITIONED STIMULUS 

15 • To say that the pigeon will emit pecks at a low rate or 
frequency Is to say that there Is a low 
of pecking the key, 

A. CONDITION 

B. RESPONSE 

C. PROBABILITY 

D. THRESHOLD 

16, A response elicited by a conditioned stimulus Is a(n) 



A. NEUTRAL STIMULUS 

B. UNCONDITIONED STIMULUS 

C. CONDITIONED RESPONSE 

D. CONDITIONED REFLEX 

17* If 3 child Is to learn to salivate to the word "candy," 
"candy" and eating candy must occur nearly , 

A. CONCURRENTLY 

B. TOGETHER 

C. JOINTLY 

D. SIMULTANEOUSLY 

l8. In the unconditioned hand-withdrawal reflex, the movement 
of the arm Is the . 

A. UNCONDITIONED RE'SFOMsE 

B. UNCONDITIONED STIMULUS 

C. UHCONDITIONED REFLEX 

D. CONDITIONED REFLEX 
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19. The more Intense the stimulus, the greater the 

of the response 

A, THRESHOLD 

B. MAGNITUDE 

T.Arm?Mrnr 

D, POWER 

20. Candy put in the mouth of the child for the first time 

salivation. 



A, EVOKES 

B, CAUSES 

C, TRIGGERS 

D, ELICITS 

21. The response is not emitted in the process called 

A. ELICITATION 

B. LATENCY 

C. CONDITIONING 

D. FORGETTING 

22. Reinforcing a response produces an Increase in the 
that a response will occur again. 

A, REINFORCEMENT 

B. THRESHOLD 

C. PROBABILITY 

D, CONDITIONING 

23. In the conditioning experiment, the of the 

conditioned response fluctuated slightly between 1 and 2 
seconds, remaining essentially constant after Trial 30. 

A. LATENCY 

B. EXTINCTION 

C. THRESHOLD 
Ds INTENSITY 

24. A kick of the leg is by a tap on the patellar 

tendon. 

A, ELICITED 

B, EMITTED 

C, INSTIGATED 

D, CAUSED 



23* An Initially neutral stimulus ceases to be able to elicit a 
response after _____ has taken place. 

A. LATENCY 

B. UNCONDITIONINO 

C. ELICITATION 

D. EXTINCTION 



26, As the number of trials In which a conditioned stimulus Is 
presented alone Increases, the (l) of a 

conditioned response decreases until (s) is 

complete. 



(1) A. EXTINCTION 

B. MAGNITUDE 

C. CONDITIONING 

D. THRESHOLD 



(2) A. LATENCY 

B. LEARNING 

C. UNCONDITIONINO 

D. EXTINCTION 



27. A given peck on the key Is a(n) 

A. STIMULUS 

B. THRESHOLD 

C. PROBABILITY 

D. RESPONSE 
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INSTRUCTIONS! Pill in the blank with the correct word or words 

to complete the statement. 



1. In the unconditioned hand -withdrawal reflex, heat is the 



2. When a peck is followed by food, the food is called a(n) 



3« The room used for conditioning experiments is designed to 

uncontrolled factors which might affect the 
experimental result. 

4. In reflex behavior, the process by which a new stimulus comes 

to elicit a response is called . 

5. As the number of trials in which a conditioned stimulus is 

presented alone Increases, the of a conditioned 

reflex Increases. 

6. In a reflex, a sufficient explanation of the response is a 

description of the preceding . 

7. Pood is not reinforcing, unless the animal has first been 

food for some time. 



8. As the number of pairings of the conditioned and unconditioned 

stimuli Increases, the _ of the conditioned reflex 

decreases until it reaches a limit. 

9. Since a tone has no effect on salivation before conditioning, 

it is a(n) . 

10. As the number of pairings of the conditioned and unconditioned 

stimuli Increases, the of the conditioned response 

Increases until it reaches a limit. 

11. The softest touch on the surface of the eye needed to elicit 

a blink marks the of the stimulus. 

12. A stimulus able to elicit a response only after conditioning 

is called a(n) . 

13« A tone and the salivation elicited comprise a(n) 

(salivary) . 

l4. If the conditioning experiment described in Set 4 had con- 
tinued to Trial 50, the of the conditioned 

response would probably have been in the vicinity of 60. 
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13* A stimulus able to elicit a response without previous 
conditioning Is called a(n) , 

16. The general behavior of "pecking the key" is a(n) 

17. behavior Is Influenced by the consequences of 
previous, similar responses. 

18. A new neutral stimulus* Is able to elicit a response after 
taken place. 

19* Since no stimuli are observed for such responses as flicking 
leaves or bar pressing, we cannot say that these responses 
are by stimuli, as are the responses In reflexes. 

20. In the knee-jerk reflex, we call the tap or blow the 

21. To condition a reflex, an Initially 

is paired with a(n) . 

22. In behavior, a stimulus precedes the response. 

23. In a reflex, the stimulus must be Intense enough to exceed 
the _______ or no response will occur. 

24. The response of turning on the electrically-operated heat 

lamp will be more frequently In the future 

If the organism is cold and hungry. 

23. The time between the tap and the kick Is the of 

the knee-jerk reflex. 

26. In the knee-jerk reflex, we call the rubber hammer the 



27. The response Is emitted without reinforcement only In the 
process called . 
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